A Deeper Look at the Hessian Eigenspectrum of Deep Neural Networks and its Applications to Regularization

Loss landscape analysis is extremely useful for a deeper understanding of the generalization ability deep neural network models. In this work, we propose layerwise loss where surface at every layer studied independently and also on how each correlates to overall surface. We study by studying eigenspectra Hessian layer. particular, our results show that geometry largely similar entire Hessian. report an interesting phenomenon eigenspectrum middle layers are observed most eigenspectrum. maximum eigenvalue trace (both full layerwise) reduce as training progresses. leverage these observations new regularizer based Penalizing indirectly forces Stochastic Gradient Descent converge flatter minima, which shown have better performance. such can be leveraged penalize middlemost alone, yields promising results. Our empirical studies well-known nets across datasets support claims work.